DLMs

This section covers the following topics:

• DLM Concepts
• How DLMs Work
• The Module Description (.dlm) File
• The IDL_Load() function
• DLM Example
• Packaging and Installing DLMs

DLM Concepts

The IDL_SYSFUN_DEF2 structure, which is described in Registering Routines, contains all the information required for IDL to compile calls to a given system routine:

• A routine signature (Name, minimum and maximum number of arguments, whether the routine accepts keywords).
• A pointer to a compiled language function (usually C) that supplies the standard IDL system routine interface (argc, argv, argk) and which implements the desired operation.

IDL does not require the actual code that implements the function until the routine is called: it is able to compile other routines and statements that reference it based only on its signature.

DLMs exploit this fact to load system routines on an “as needed” basis. The routines in a DLM are not loaded by IDL unless the user calls one of them. A DLM consists of two files:

1. A module description file (human readable text) that IDL reads when it starts running. This file tells IDL the signature for all system routines contained in the loadable module.
2. A sharable library that implements the actual system routines.This library must be coded to present a specific IDL mandated interface (described below) that allows IDL to automatically load it when necessary without user intervention.

DLMs are a powerful way to extend IDL’s built-in system routines. This form of packaging offers many advantages:

• Unlike LINKIMAGE, IDL automatically discovers DLMs when it starts up without any user intervention. This makes them easy to install; you copy the two files into a directory on your system where IDL will look for them.
• DLM routines work exactly like standard built-in routines and are indistinguishable from them. There is no need for the user to load them (for example, using LINKIMAGE) before compiling code that references them.
• As the amount of code added to IDL grows, using sharable libraries in this way prevents name collisions in unrelated compiled code from fooling the linker into linking the wrong code together. DLMs thus act as a firewall between unrelated code. For example, there are instances where unrelated routines both use a common third party library, but they require different versions of this library. A specific example is that the HDF support in IDL requires its own version of the NetCDF library. The NetCDF support uses a different incompatible version of this library with the same names. Use of DLMs allows each module to link with its own private copy of such code.
• Since DLMs are separate from the IDL program, they can be built and distributed on their own schedule independent of IDL releases.
• System routines packaged as DLMs are effectively indistinguishable from routines built-into IDL.

How DLMs Work

IDL manages DLMs in the following manner:

1. When IDL starts, it looks in the current working directory for module definition (.dlm) files. It reads any file found and adds the routines and structure definitions defined to its internal routine and structure lookup tables as “stubs”. In the system routine dispatch table, stubs are entries that inform IDL of the routine’s existence, but which lack an actual compiled function to call. They contain sufficient information for IDL to properly compile calls to the routines, but not to actually call them. Similarly, stub entries in the structure definition table allow IDL to know that the DLM supplies the structure definition, but the actual definition is not present.

   After it looks in the current working directory, IDL searches !DLM_PATH for .dlm files and adds them to the table in the same manner. The default value of !DLM_PATH is the directory in the IDL distribution where the binary executables are kept (bin/bin.platform). This default can be changed by defining the IDL_DLM_PATH preference (this is similar to the way the IDL_PATH preference works with !PATH).

   Note: If you redefine the IDL_DLM_PATH preference, be sure to include the <IDL_DEFAULT> token. IDL will not run correctly if the default DLM directories are not included in !DLM_PATH.

   See Packaging and Installing DLMs for additional information about how IDL selects sharable libraries on different platforms.

2. The IDL session then continues in the usual fashion until a call to a routine from a loadable module occurs. At that time, the IDL interpreter notices the fact that the routine is a stub, and loads the sharable library for the loadable module that supplies the routine. It then looks up and calls a function named IDL_Load(), which is required to exist within your library. The IDL_Load's job is to replace the stubs from that module with real entries (using IDL_SysRtnAdd) and otherwise prepare the module for use.
3. Once the module is loaded, the interpreter looks up the routine that caused the load one more time. If it is still a stub then the module has failed to load properly and an error is issued. Normally, a full routine entry is found and the interpreter successfully calls the routine.
4. At this point the module is fully loaded, and cannot be distinguished from a compiled part of IDL. A module is only loaded once, and additional calls to any routine, or access to any structure definition, from the module are made immediately and without requiring any additional loading.

The Module Description (.dlm) File

The module description file is a simple text file that is read by IDL when it starts or when the !DLM_PATH variable is changed. Module description files have the file suffix .dlm. The information in the .dlm file tells IDL everything it needs to know about the routines supplied by a loadable module. With this information, IDL can compile calls to these routines and otherwise behave as if it contains the actual routine. The loadable module itself remains unloaded until a call to one of its routines is made, or until the user forces the module to load by calling the DLM_LOAD procedure.

Empty lines are allowed in DLM files. Comments are indicated using the # character. All text from a # to the end of the line is a comment and ignored by IDL.

All other lines start with a keyword indicating the type of information being conveyed, possibly followed by arguments. The syntax of each line depends on the keyword. Possible lines are:

MODULE Name

Gives the name of the DLM. This should always be the first non-comment line in a .dlm file. This is the name used by DLM_LOAD to manually load the module and is displayed with HELP, /DLM. There can only be one MODULE line.

MODULE JPEG

DESCRIPTION DescriptiveText

Supplies a short one line description of the purpose of the module. This information is also displayed by HELP, /DLM. This line is optional.

DESCRIPTION IDL JPEG support

VERSION VersionString

Supplies a version string that can be used by the IDL user to determine which version of the module will be used. IDL does not interpret this string, it only displays it as part of HELP, /DLM. This line is optional.

VERSION 6a

BUILD_DATE DateString

If present, IDL will display this information as part of the output from HELP, /DLM. This line is optional.

BUILD_DATE JAN 8 1998

SOURCE SourceString

A short one line description of the person or organization that is supplying the module. This line is optional.

SOURCE My Company Name

CHECKSUM CheckSumValue

This directive is used internally by IDL to sign the authenticity of the DLMs included within the IDL release. It is not required for user-written DLMs.

STRUCTURE StructureName

There should be one STRUCTURE line in the DLM file for every named structure definition supplied by the loadable module. If you refer to such a structure before the DLM is loaded, IDL uses this information to cause the DLM to load. The IDL_Init() function for the DLM will define the structure.

GLOBAL_SYMBOLS

This line is optional. Including this line in the DLM file will cause the shared library to load all of its symbols (functions or procedures) as globally accessible rather than locally accessible. If a symbol is globally accessible, then libraries that are loaded later will be able to access the symbol. In practice, adding this line to the DLM file will cause IDL to set the RTLD_GLOBAL flag when calling the dlopen() operating system function to load the module.

On Microsoft Windows and macOS systems, symbols are automatically loaded as global. A GLOBAL_SYMBOLS line in the DLM file will be quietly ignored.

Use caution when making a DLM’s symbols globally accessible. Judicious naming of the DLM’s symbol names will help ensure that symbols exported by the DLM will not cause namespace collisions with symbols from other libraries.

FUNCTION RtnName [MinArgs] [MaxArgs] [Options...]

PROCEDURE RtnName [MinArgs] [MaxArgs] [Options...]

There should be one FUNCTION or PROCEDURE line in the DLM file for every IDL routine supplied by the loadable module. These lines give IDL the information it needs to compile calls to these routines before the module is loaded.

RtnName

The IDL user level name for the routine. The routine name can be a simple procedure or function name (e.g. MY_PROCEDURE or MY_FUNCTION), or the name of an object method (e.g. MY_OBJECT::PROCEDURE_METHOD or MY_OBJECT::FUNCTION_METHOD).

MinArgs

The minimum number of arguments accepted by this routine. If not supplied, 0 is assumed.

MaxArgs

The maximum number of arguments accepted by this routine. If not supplied, 0 is assumed.

Options

Zero or more of the following:

OBSOLETE: IDL should issue a warning message if this routine is called and !WARN.OBS_ROUTINE is set.

KEYWORDS: This routine accepts keywords as well as plain arguments. For example, a procedure named READ_JPEG that accepts a minimum of one argument, a maximum of three arguments, and also accepts keyword arguments would have the following definition in the .dlm file:

PROCEDURE READ_JPEG 1 3 KEYWORDS

The IDL_Load() function

Every loadable module's sharable library must export a single symbol called IDL_Load(). IDL calls this function when it loads the module and is expected to do all the work required to load real definitions for the routines supplied by the function and prepare the module for use. This always requires at least one call to IDL_SysRtnAdd(). It usually also requires a call to IDL_MessageDefineBlock() if the module defines any messages. Any other initialization needed would also go here:

int IDL_Load(void)

This function takes no arguments. It returns True (non-zero) if it was successful, and False (0) if an initialization step failed.

DLM Example

This example creates a loadable module named TESTMODULE. Code for this example is included in the external/dlm subdirectory of the IDL

installation. TESTMODULE provides 2 routines:

TESTFUN

A function that issues a message indicating that it was called, and then returns the string “TESTFUN” This function accepts between 0 and IDL_MAXPARAMS arguments, but it does not use them for anything.

TESTPRO

A procedure that issues a message indicating that it was called. This procedure accepts between 0 and IDL_MAX_ARRAY_DIM arguments, but it does not use them for anything.

The intent of this example is to show the support code required to write a DLM. This framework can be easily adapted to real modules by replacing TESTFUN and TESTPRO with other routines.

The first step is to create the module definition file for TESTMODULE, named testmodule.dlm:

MODULE testmodule

DESCRIPTION Test code for loadable modules

VERSION 1.0

SOURCE NV5 Geospatial Solutions

BUILD_DATE JAN 8 1998

FUNCTION TESTFUN 0 IDL_MAXPARAMS

PROCEDURE TESTPRO 0 IDL_MAX_ARRAY_DIM

The next step is to write the code for the sharable library testmodule.c:

 

#include <stdio.h>

#include "idl_export.h"

 

/* Define message codes and their corresponding printf(3) format strings. Note that message codes start at zero and each one is one less that the previous one. Codes must be monotonic and

contiguous. */

 

static IDL_MSG_DEF msg_arr[] = {

#define M_TM_INPRO 0

{ "M_TM_INPRO", "%NThis is from a loadable module procedure.” },

 

#define M_TM_INFUN -1

{ "M_TM_INFUN”, "%NThis is from a loadable module function.” },

};

 

/* The load function fills in this message block handle with the opaque handle to the message block used for this module. Th other routines can then use it to throw errors from this

block. */

 

static IDL_MSG_BLOCK msg_block;

 

/* Implementation of the TESTPRO IDL procedure */

static void testpro(int argc, IDL_VPTR *argv)

{ IDL_MessageFromBlock(msg_block, M_TM_INPRO, IDL_MSG_RET); }

 

/* Implementation of the TESTFUN IDL function */

static IDL_VPTR testfun(int argc, IDL_VPTR *argv)

{

IDL_MessageFromBlock(msg_block, M_TM_INFUN, IDL_MSG_RET);

return IDL_StrToSTRING("TESTFUN");

}

 

int IDL_Load(void)

{

/* These tables contain information on the functions and procedures

that make up the TESTMODULE DLM. The information contained in these tables must be identical to that contained in testmodule.dlm.

*/

 

static IDL_SYSFUN_DEF2 function_addr[] = {

{ testfun, "TESTFUN”, 0, IDL_MAXPARAMS, 0, 0},

};

static IDL_SYSFUN_DEF2 procedure_addr[] = {

{ (IDL_SYSRTN_GENERIC) testpro, "TESTPRO”, 0, IDL_MAX_ARRAY_DIM, 0, 0},

};

 

/* Create a message block to hold our messages. Save its handle where the other routines can access it. */

 

if (!(msg_block = IDL_MessageDefineBlock("Testmodule”,

IDL_CARRAY_ELTS(msg_arr),

msg_arr))) return IDL_FALSE;

 

/* Register our routine. The routines must be specified exactly the same as in testmodule.dlm. */

 

return IDL_SysRtnAdd(function_addr, TRUE,

IDL_CARRAY_ELTS(function_addr))

&& IDL_SysRtnAdd(procedure_addr, FALSE,

IDL_CARRAY_ELTS(procedure_addr));

}

 

If building a DLM for Microsoft Windows, a linker definition file (testmodule.def) is also needed. All of these files, along with the commands required to build the module can be found in the dlm subdirectory of the external directory of the IDL distribution.

Once the loadable module is built, you can cause IDL to find it by doing one of the following:

• Move to the directory containing the .dlm and sharable library for the module.
• Define the IDL_DLM_PATH preference to include the directory. Running IDL to demonstrate the resulting module:

IDL> HELP,/DLM,’testmodule’

 

** TESTMODULE - Test code for loadable modules (not loaded) Version:1.0,Build Date:JAN 8 1998,Source:NV5 Geospatial Solutions.

Path: /home/user/testmodule/external/testmodule.so

 

IDL> testpro

 

% Loaded DLM: TESTMODULE.

% TESTPRO: This is from a loadable module procedure. IDL> HELP,/DLM,’testmodule’

** TESTMODULE - Test code for loadable modules (loaded) Version:1.0,Build Date:JAN 8 1998,Source:NV5 Geospatial Solutions.

Path: /home/user/testmodule/external/testmodule.so

 

IDL> print, testfun()

 

% TESTFUN: This is from a loadable module function.

TESTFUN

The initial HELP output shows that the module starts out unloaded. The call to TESTPRO causes the module to be loaded. As IDL loads the module, it prints an announcement of the fact (similar to the way it announces the .pro files it automatically compiles to satisfy calls to user routines). Once the module is loaded, subsequent calls to HELP show that it is present. Calls to routines from this module do not cause the module to be reloaded (notice that calling TESTFUN did not cause an announcement message to be issued).

Packaging and Installing DLMs

Once you have created sharable library (.so or .dll) and module description (.dlm) files, you will need to ensure that the files are installed in a location where IDL can find and load the libraries.

Single-Platform DLMs

If your module will be installed only on computers of a single architecture, use this process:

1. Create the sharable library file. The file will have the extension .dll for Microsoft Windows platforms, or .so for UNIX-like platforms (Mac and Linux).
2. Create the module description file (.dlm) as described in “The Module Description (.dlm) File” (above).
3. Place both the sharable library file and the module description file in a directory included in IDL’s IDL_DLM_PATH preference. See “Installing DLMs Using the IDL Workbench Update Mechanism” for additional notes.
4. Restart IDL.

Even if your module supports only one platform, consider following the naming rules described in “How IDL Selects the Correct Sharable Library File”. Using the multi-platform naming rules incurs no performance penalty, and may save effort if you end up supporting other platforms in the future.

Multi-Platform DLMs

If your module will be installed on computers of different architectures, you must create a unique sharable library file for each architecture. To install the DLM on a user’s machine, you have the following options:

Create Platform-Specific Installations

If you create a separate installation package for each architecture, creating a multi- platform DLM is essentially just creating a series of Single-Platform DLMs, one for each platform. Use caution with this approach, since you will have to ensure that if your end-user installs more than one platform’s version of the DLM, the module description and shared library files for the different platforms are installed in the correct directories.

Create a Multi-Platform Installation

You can create a single installation package that supports multiple architectures if you follow a simple set of naming rules when creating your sharable library files. To create a multi-platform installation package:

1. Create a sharable library file for each platform, following the naming rules described in “How IDL Selects the Correct Sharable Library File” (below).
2. Create a single module description file (.dlm) as described in “The Module Description (.dlm) File”.
3. Place the module description file and all of the sharable libraries in a single directory included in IDL’s IDL_DLM_PATH preference. See “Installing DLMs Using the IDL Workbench Update Mechanism” for additional notes.
4. Restart IDL.

How IDL Selects the Correct Sharable Library File

When IDL starts, it searches for DLMs in the directories included in IDL’s IDL_DLM_PATH preference as described in “How DLMs Work” (above). When IDL finds a module description file, it adds the routines and structure definitions defined by the DLM to its internal routine and structure lookup tables.

It is not until later, when a user calls a routine defined by the DLM, that IDL actually loads the sharable library. At this point, IDL searches for a sharable library file built for the current platform.

IDL uses the following process to search for the sharable library file:

1. IDL constructs the base name of the library file by removing the .dlm suffix from the module definition file’s name.

2. To the library’s base name, IDL appends a platform-specific string. The specific strings are shown in the table below. The string is the concatenation of the name of the platform’s platform-specific bin subdirectory, along with the suffix .dll on Windows systems or .so on all UNIX-based systems.

   For example, if the name of the DLM file is

   my_module.dlm

   then the platform-specific sharable library file name for a 64-bit Linux platform would be

   my_module.linux.x86_64.so

3. IDL searches the directory that contains the module definition file for a library file with the platform-specific sharable library file name. If it finds a matching file, it loads the library and executes the routine called by the user.

4. If IDL does not find the platform-specific library file, it searches the directory that contains the module definition file for a library file with the same base name as the module definition file, replacing the .dlm extension with the suffix .dll or .so.

   For example, if the name of the DLM file is

   my_module.dlm

   then the generically-named sharable library file name would be

   my_module.dll

   on a Windows system, or

   my_module.so

   on a UNIX system.

   If IDL finds a generically-named sharable library file, it attempts to load the library and execute the routine called by the user. Note that IDL will only be able to successfully load the library if the generically-named library file was built for the current platform.

5. If IDL fails to find either the platform-specific sharable library file or the generically-named library file, it will issue one of the following error messages:

   If a platform-specific sharable library file for a different platform exists in the same directory, the error message is

   Dynamically loadable module is unavailable on this platform:

   my_module.

   If no platform-specific sharable library files for any platform are present, the error message is

   Dynamically loadable module failed to load: my_module.

   The first message indicates that the DLM exists but is not supported for the current platform, the second indicates that the DLM does not exist, despite the presence of the .dlm file.

   One benefit of this file naming and search procedure is that you can distribute a DLM package that includes library files for several platforms in a single directory. IDL will load the correct shared library for the end-user’s platform, or provide a sensible error message if the platform is not supported.

Platform-Specific Sharable Library File Suffixes

The following table lists the platform-specific file suffixes for IDL’s supported platforms:

Windows 64-bit: .x86_64.dll

Linux 64-bit: .linux.x86_64.so

macOS Intel 64-bit: .darwin.x86_64.so

Example DLM Distribution

For example, suppose you have created a dynamically loadable module named my_cool_module, and created sharable libraries for Windows (32- and 64-bit) and Linux (64-bit) but not for macOS. Your DLM installation directory would contain the following files:

my_cool_module.dlm

my_cool_module.x86.dll

my_cool_module.x86_64.dll

my_cool_module.linux.x86_64.so

If a user on a macOS system attempts to call a routine from the my_cool_module DLM, the fact that sharable libraries for other platforms exist informs IDL that the DLM intentionally does not provide support for that platform. If a user on one of these unsupported platforms attempts to use the functionality from the DLM, IDL will issue the message

Dynamically loadable module is unavailable on this platform:

my_cool_module.